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ABSTRACT 



The education of students in the techniques of critical 
appraisal of drug studies has been identified as a deficiency in many health 
sciences curricula. Errors in research design and inconsistencies in the 
reporting of study results persist in professional pharmacy and medical 
journals. Thus, thorough and accurate review and interpretation of journal 
studies are essential for assuring that patients receive proper drug therapy. 
This project developed an interactive, computer instructional program that 
would teach students to evaluate all aspects of published drug efficacy 
studies. The content for the program was identified, prepared, and designed 
for an interactive computer format, and the program was developed using 
"Authorware Professional." The program, "Evaluation of Clinical Drug 
Studies," consists of 10 main sections with several subsections. Two 
consecutive classes of pharmacy students at West Virginia University tested 
the program, which was found to increase significantly their knowledge of 
critical drug study evaluation techniques compared to a control group and 
pretests. Appendices include the timetable for the project; "Development and 
Evaluation of a Computer-Assisted Instructional Program To Teach Critical 
Evaluation of Drug Studies" (Marie A. Abate, Arthur I. Jacknowitz, James M. 
Shumway, and Anne H. Nardi)--an article published in "American Journal of 
Pharmaceutical Education," Volume 57, winter 1993; computer program 
evaluation summary for 1994; and information for FIPSE. (SWC) 
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Summary 






The education of students in the techniques of critical appraisal of drug 
studies has been identified as a deficiency in many health sciences curricula. 
Errors in research design and inconsistencies in the reporting of study results 
persist in professional pharmacy and medical journals. Thus, thorough and 
accurate review and interpretation of journal studies are essential for assuring 
that patients receive proper drug therapy. The objective of this project was to 
develop an interactive, computer instructional program that would teach students 
to evaluate all aspects of published drug efficacy studies. The content for the 
program was identified, prepared, and designed for an interactive computer 
format, and the program was developed using Authorware Professional®. The 
program. Evaluation of Clinical Drug Studies,” consists of 10 main sections with 
several subsections. Two consecutive classes of pharmacy students tested the 
program, which was found to increase significantly their knowledge of critical 
drug study evaluation techniques compared to a control group and pretests. The 
computer program has recently been converted into an IBM Windows format, in 
addition to the original Macintosh format. Plans are currently underway to 
distribute the program to interested schools and colleges of pharmacy and 
medicine, health care practitioners, and pharmaceutical industry professionals. 

Marie A. Abate, Pharm.D. 

West Virginia University 
School of Pharmacy 
P.O. Box 9520 

Morgantown, WV 26506-9520 
(304) 293-1463 

The “Evaluation of Clinical Drug Studies” program received one of the 
“Innovations in Teaching” awards from the American Association of Colleges of 
Pharmacy in 1993. 

Results from program testing were published: 

Abate MA, Jacknowitz AI, Shumway JM, Nardi AH. Development and 
Evaluation of a Computer-Assisted Instructional Program to Teach Critical 
Evaluation of Drug Studies. Am J Pharm Educ 1993;57:416-424. 

The “Evaluation of Clinical Drug Studies” program received U.S. copyright 
registration in August, 1994. 
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Executive Summary 

Development of a Computer System to Educate Students 
to Evaluate and Interpret Published Drug Studies 

West Virginia University 
School of Pharmacy 
P.0. Box 9520 

Morgantown, WV 26506-9520 
Marie A. Abate, Pharm.D. 

(304)293-1463 



Project Overview 

A computer instructional program was developed to teach health professions students to 
critically analyze and interpret published drug efficacy studies. An initial version of the program, 
“Evaluation of Clinical Drug Studies,” was developed over a 16 month period for use on the 
Macintosh computer. It was evaluated by second professional year pharmacy students at WVU 
during two consecutive years (1993 and 1994). The program was found to significantly increase 
students’ knowledge of drug study evaluation concepts and to improve their ability to critically 
evaluate published efficacy studies. Overall, students felt that the program was of high quality and 
useful in increasing their subject knowledge. An IBM compatible Windows version of the 
program is currently being completed. The most appropriate avenue(s) to use for program 
dissemination are being identified and should be initiated shortly. 

Purpose 



The idea to develop a computer instructional program to teach critical evaluation of 
published drug studies arose due to a number of factors. First, the importance of literature 
evaluation skills was well known to the investigators, two of whom are drug information 
specialists. Second, computer assisted instruction was incorporated into pharmacy school 
curricula to only a limited extent nationwide. However, the flexibility and portability of computer 
assisted instruction, combined with its student centered learning focus, made it desirable to 
develop. Third, although a number of checklists, scoring systems and algorithms were published 
to assist health care providers in evaluating clinical studies, they were generally brief and did not 
adequately explain the concepts contained. Finally, pharmacy students at WVU received a required 
one credit hour course in drug literature, but the course focused on texts and not journal articles. 
This, combined with the investigators’ interest in computers and instructional methods, led to the 
idea of developing a comprehensive computer program to teach the critical analysis of drug efficacy 
studies. 



Pitfalls encountered with the development and use of the computer program include the 
following. It required a considerable amount of time to learn how to use Authorware 
Professional®, more than originally anticipated. An instructional designer should also be available 
to anyone who initiates a project such as this. Adequate time must be allowed to identify the 
area(s) of the curriculum in which to utilize such a computer program. Finally, adequate 
equipment and private work areas must be available for student use. 

Background and Origins 

Computer assisted instruction was used infrequently at the West Virginia University 
Schools of Pharmacy and Medicine prior to initiation of this project. Another curricular deficiency 
was that the students did not receive any formal education in drug literature evaluation techniques, 
although the need for such education has been increasingly called for in recent years. There are 
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many types of published drug literature, including efficacy, safety, pharmacokinetic and 
pharmacoeco„om,c studies. It was decided to focus progim devdopS“„ dZ £ey since 

Th W ° U d UU lZe these studies most fre quently and a broad range of concepts could be 
development ini,ia “ y Supportive otth ° !»**• “<* »"A-d s“SgCi« 

Project Descrintinii 

Computer Program Development and Timetable 

itoSth n,id :s epttn > | *™ 8 99 t i 1 

SltffiSfW 1 ""* ^ “^8- PreparattouVthe conmmZte 

program included ideas gathered taVsXFbXanlj^ 

were used by the investigate* for contem and des£t S aS SoftShTSe, 
the development and testing of the computer progrL is found in Appendix B 

Computer Program Use and Evaluation 

c#»r*raTiH Dudngthe Spring 1993 semester (January - May), the computer program was used bv the 

SSoESMr ph T a Cy t S, “u dentS < ? 6 t0al) in a »**•« credit ff •SSSS m 
course - Each week, the students working in groups were assigned a different 

the computer program t £5 omp,ete on their own The subject matter was entirely self-contained in 
* . °T r P r °gram. The Program automatically recorded the amount of time that each 

^he^omputer 1 was compiled ' C ° mPU ‘ er ’ a " d ‘ he ' 0,al an ’ OU "* ° f time s,uden,s a< =“ al1 X spent using 
SUSSEX?* literature evaluadon mahTrial as & 

educat on 1 his pre-test was very similar in content to the pre-test given later to the studv grmm of 
students A post-test, similar in content to that of the pre-test but lfnger hr length was ^ P 
to f the contrast group students during their final class in May 1992 Computer 

L 5 f eV f?° n da if were a,so obtained from the pharmacy students in the expanded 2 credit 
hour Introduction to Drug Literature” course taught during the Spring 1994 semester. A Dre-te<t* 

collected. CS 3 S ° administered to these students, andcomputer use and attitudinal data were 
Program Revisions and Updates 

nno P ro gram has undergone considerable changes since the time it was first used Minor 
mvi™ Pr ° §ram conte J nt have been made continuously, based upon student feedback ’ 

used i„ Windows* th^Win^owsv^rsioifis^now S' t m^eS aCCOmm0date *“ » 
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cf.,Hv „ The pre ' t ? s [ results ’ post-test results, and attitude survey findings from the contrast anH 
the mean and medSn ores' tot h??^™ 8r ° UP ’ a described ■" Appendix C. Both 

ou, of » tea, po,r StwThT^g^S °£Z2Tx%,Z, ™TZr t21 

pre-test score for the study group students was 44%, wifh a ranee of 5 S? ) ' The mean 

test score for the study group was 77%, with a ranoe of 51 % 97% Thk 7 56% * The mean poSt ' 
stattstically significantly different fmm both the pre test score ajwell as the"^ 6 ? 6 7“ 

= 58% 98% 'i Thp • ’ ? mparec * to an average of 83% on the post-test (median = 85% ranoe 

effectiveness in " d reinf0rCeS ' he Pint's 

The study group students spent a mean of approximately 1 1 hours franco - a r. \ 
computer program (median = 10 4 hours ranoe = 1 ? P fn I ho.. ' \ ^ mean ± SD ) usin § the 

beT success.' “ “ StUdent ° pini ° n ’ ,he P ro S ram and P ro j ecl “ a whole have been deemed to 

Plans for Continuation and Dissemination 

The computer program will continue to be used in the WVU School of Pharmacv’s 

manuf 8l f^^^ S ^ d ^^ d ^^ 7 ^'^ a7> ^^ en ^ n ' a '^^S S ^ori^g with^phamacetnkai 106 
m»nu£ Urer 'a 'I 5 W 1 distributio " ^otts. Initial discussion and neglahon w™h a 
manufacturer, Astra/Merck, is underway. The principal investigator also intends to «ain oreater 

c P otpme n r C p y mgmm e "" ° f Autbo ™- Sessional® in order to facilitate her woSlT 

Summ ary and Conclusions 
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Final Report - Main Section 



Project Overview 

A computer instructional program was developed to teach health 
professions students how to critically analyze and interpret published drug 
efficacy studies. The project was initiated because students at West Virginia 
University (WVU) did not receive any instruction in this area, and increased 
computer use at the WVU School of Pharmacy was desired. Also, the content 
was well-suited for development in a computerized format since the concepts 
involved were not expected to change rapidly. Thus, program updating would be 
easier. 



An initial version of the computer program, “Evaluation of Clinical Drug 
Studies,” was developed over a 16 month period for use on the Macintosh 
computer. It was evaluated by second professional year pharmacy students at 
WVU during two consecutive years (1993 and 1994). The program was found to 
significantly increase students knowledge of drug study evaluation concepts and 
to improve their ability to critically evaluate published efficacy studies, compared 
to pre-tests and a contrast group of students. In general, students felt that the 
program was of high quality and useful in increasing their subject knowledge. 

The major criticisms related to the time involved to complete the program 
(thought by some to be too much for a 1 or 2 credit hour class), the lack of 
detailed handout material (remedied to an extent by the development of a fairly 
comprehensive workbook containing a content outline and learning objectives), 
and the lack of need for undergraduate pharmacy students to know such material 
(certain students expressed this concern, even though drug study evaluation skills 
are deemed to be essential by educators and clinical practitioners nationally). 

Overall, the project was shown to be a success. An IBM compatible 
Windows version of the program is currently being completed. The most 
appropriate avenue(s) to use for program dissemination are being identified and 
should be initiated shortly. 



Purpose 



The idea to develop a computer instructional program to teach critical 
evaluation of published drug studies arose due to a number of factors. First, the 
importance of literature evaluation skills was well known to the investigators, two 
of whom are drug information specialists. An American Association of Colleges 
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of Pharmacy (AACP) Background Paper identified the ability to appraise and 
evaluate cl, meal drug studies as a skill required of entry-level pharmacy 
practitioners in order to provide pharmaceutical care. The need for pharmacy 
and med'ea students and practitioners to possess these skills was also being 
widely publicized in a variety of professional journals. S 

school“;i,T P r a f SiSted , inStrUCtion was incorporated into pharmacy 
school curr cula to only a limited extent nationwide. However the flexibility 

and portability of computer assisted instruction, combined with its student * 

rs. r'" g r foeus ’ r de i,de r bie to devei °p- ™ rd - ^7 number 

ot checklists, scoring systems and algorithms were published to assist health care 
providers m evaluating clinical studies, they were generally brief and did not 
adequately explain the concepts contained. It was anticipated that a computer 
instructional program would be a useful method for teaching drug Sure 
™'“™ techniques, since many examples and levels of explanation could be 
incorporated into a computer and reviewed by users to the extent desired. 

„„„„ F,na , ll >’’ Pharmacy students at WVU received a required one credit hour 
course in drug literature, but the course focused on texts and not journal articles 

meSstdm tfS com P ute * and iZuctiont 

rndhods, led to the idea of developing a comprehensive computer program to 

teach the critical analysis of drug efficacy studies. P 8 

The program was originally anticipated to serve as a completely self- 

fn 0 ^SS lr h C h° nal P T ram ’ i e " With ° Ut 3 need to P^ide content lectures. 
In this regard, it has served its purpose reasonably well. However, to help 

c!iihcarstudies t ^The nCePtS learned ’ “T? 65 Were develo P ed based upon actual 
clinical studies. The exercises were useful for determining areas of confusion or 

misunderstanding in the computer program’s content; this allowed for 

Clanficahon Of these areas and the incorporation of additional examples. About 

10 - 15 minutes of each weekly class were also allocated for answering student 

Z'l995 a nnlhI C ? nt H nt ' alth ° Ugb StUdentS did not usuall >' ask questions This 

periods^eL sir h L'd f dUC ‘° DrUg L,lerature cours<; - four 50 minute class 
periods were set aside for computer program content clarification and 

to tZdrnZZ' Add,t, ° n l ClaSS periods were also “ sed to review the answers 

the necessfreZh ff rCISeS ' Z , T S “ g3t ° rS 3re n °‘ convmced ' however, about 
he necessity of this from a student learning perspective. In the future a self- 

^ cZ^pregmm.” ^ C ° U ' d be deVe '° ped f ° r USe in co "J a " c “ 0 " whh 
Pitfalls encountered with the development and use of the computer 
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program include the following. It required a considerable amount of time to 
learn how to use Authorware Professional®, more than originally anticipated. 
Since the investigators had other responsibilities in addition to this project, it was 
difficult at times to keep on schedule and the principal investigator often had to 
pi ck up the workload of others. A principal investigator needs to understand this 
and be prepared to expend the extra energy required to make a project work. An 
instructional designer should also be available to anyone who initiates a project 
such as this. They would be extremely valuable in designing the screen layouts, 
selecting fonts and colors, and in helping to use software such as Authorware 
Adequate time must be allowed to identify the area(s) of the curriculum in which 
to utilize such a computer program. Although it was anticipated that the program 
would be used by the medical as well as the pharmacy school, curriculum 
revision delays in the medical school have slowed the program’s introduction into 
its curriculum. Finally, adequate equipment and private work areas must be 
available for student use. Some students are distracted by crowds or talking in a 
computer lab and don’t work well under these conditions. 



Background and Origins 

Computer assisted instruction was used infrequently at the West Virginia 
University Schools of Pharmacy and Medicine prior to initiation of this project 
Individual schools within the WVU Health Sciences Center had limited resources 
for the purchase and maintenance of computer equipment, and they had 
inadequate numbers of machines to accommodate students. The need to increase 
computer use within the health sciences curricula was recognized, however; the 
WVU Health Sciences Center began operation of its Computer Based Learning 
Center (CBLC) in 1991. s 



Another curricular deficiency noted was that students in the West Virginia 
University Schools of Pharmacy and Medicine did not receive any formal 
education in drug literature evaluation techniques. The need for such education 
has been increasingly called for in recent years, by such groups as educators, 
practitioners, journal editors, and even the lay public. Thus, the investigators 
decided to create a project team to develop a computer instructional program that 
would educate students to critically evaluate the published literature. 

There are many types of published drug studies, including efficacy, safety, 
pharmacokinetic, and pharmacoeconomic studies. It was decided to focus 
program development on drug efficacy, since practitioners would utilize these 
studies most frequently and a broad range of concepts could be covered by such a 
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focus. There was also an existing course in drug literature that was taught by one 
of the investigators, which would facilitate incorporation of the completed 
computer program into the pharmacy curriculum. Another investigator is the 
Associate Dean for Curricular Affairs in the WVU School of Medicine, a position 
from which program incorporation into the medical school curriculum could be 
facilitated. 

The institution was initially supportive of the project, and remained so 
throughout its development. A commitment for support was obtained from the 
CBLC, and the Deans of the Schools of Pharmacy and Medicine. The faculty of 
the School of Pharmacy was in the process of reviewing aspects of its curriculum 
at the time of the project’s inception and was supportive of a 1 credit hour 
increase in the introductory drug literature class to incorporate the completed 
computer program. This credit hour increase was in place for the second test 
group of pharmacy students. However, incorporating the computer program into 
the medical school’s curriculum has not gone as smoothly. A major problem 
with the medical curriculum is finding a location for new material; ideally, other 
information should first be deleted. Planned curricular revisions have also been 
slower to develop than originally anticipated. When curricular changes are 
finally initiated, the investigators are optimistic that an appropriate location will 
be identified for the computer program, most likely during the second year of 
medical school. 



Project Description 

Computer Program Development and Timetable 

Authorware®, an authoring tool that allows for the development of 
instructional material incorporating sound, graphics, and animation, was used for 
the preparation of the program "Evaluation of Clinical Drug Studies" on the 
Macintosh computer. Approval to utilize this software was granted by FIPSE 
prior to funding of the original application. Key factors influencing the 
selection of Authorware® included its user-friendliness, the lack of a need for 
programming skills in order to use the software, and the future availability of 
Authorware® for Windows, which would allow the program to be fairly readily 
converted to run on Windows machines after completion of the Macintosh 
version. The StuffltTM Installation Technologies software package was used to 
compress the completed Macintosh program onto floppy disks (due to the size of 
the program — currently over 8 MB) and automatically uncompress it later. 
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J he . flrs * ste P of the P r °j ect initiation began in mid-September 1991 A 
three-day Authorware® training program was provided at this time for the A 

S“f~ 8 D 0rS by fh a ,rainin A aDd devel ° pment s P ec ' abst from the software’s 
manufacturer. During this workshop, the investigators learned the basics of 

fh "EXtLTfr ° n ' r°' l0W '" g ' he lining, preparation ofle c^ffor 
the bvaluation of Clinical Drug Studies" program began. The focus of this 

pr °f. ra . m ! n }' olve * ‘ eachm 8 health professional students how to critically evaluate 
published drug efficacy studies. The detailed content outline and learning 
bjectives for each section of the computer program are found in Appendix A - 
Computer Program Workbook. The first group of pharmacy students to use the 

to use RatoTese d st n rt 8 *!“ Sprin 8 1 l9 ? 3 semester . did Jt have this workbook 
to use. Kather, these students received a less detailed outline of topics and 

learning objectives to use in conjunction with the program. The more 

comprehensive workbook was subsequently prepared in response to the students’ 
comments that such a workbook would be beneficial. students 

The content outline for the computer program included ideas gathered 
from several books and journal articles. Each investigator was responsible for 

dSmfd^oThe °L ,he . ,ext for differ f ent contel « sections, which were then 
srhlnT!? 1 m other 1 ' nvest igators for review and revision. The investigators 
scheduled weekly meetings to facilitate content development and the discussion of 

Changes. During the latter part of 1992, a consultant reviewed^ jhe ^ to section of 
the computer program with respect to its design and interactivity. He provided 
e investigators with written feedback and suggestions for change Another 
consultant later reviewed the statistics section of the program when completed 
with regard to content; he similarly provided comments and suggestions^ 

program dunne ST ul ^ n 7 eviewed '*f c °™Plete first version of the computer 
ZeriaTs from al ,? Pr ° V ' ded Wnt,en comments and feedback. The 

Ts nn ev^anH C0 " Sul !! nts u were used b V «he investigators for content and 
design review and revision. To obtain additional computer program design ideas 

the investigators reviewed interactive educational software at the Nationaf 
1992. ry ° f MedlCme S beaming Center for Interactive Technology during May 

The timetable used for the development and testing of the computer 
program is found in Appendix B. Initially, it was planned to have a completed 
version of the computer program ready by May 1992; however, the extensiveness 
To n prf ro S ram necessitated that greater development time be spent during year 2 
To perform basic functions with Authorware®, such as entering text settfng 

ferT.SoS';" 8 feed H Ck ’ and Set,ing Up baSiC " the is 

rainy straightforward to use. However, when developing many sections. 
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branching, sound and animation, it becomes more complicated and time- 
consuming to work with. There is definitely a learning curve that must be 
navigated to optimally use Authorware®. It is important to allow sufficient time 
for actual software use and the incorporation of the developed materials. Also, 
the amount of time needed to adapt the content from written blocks of text to a 
computer screen format should not be underestimated. An instructional designer, 
not available to the investigators, would have been of immense benefit in the 
program s preparation. Also, although it was planned to field test the program 
from October to December 1992 with selected pharmacy and medical students, 
this was not possible since development and formatting of the program was still 
underway. However, a senior pharmacy student assisted with the review and 
development of the program during the fall semester (September - December, 
1992). She completed each section of the program and recorded the time 
required, prepared written comments related to the program's clarity and format, 
and helped write content and quiz questions for some of the sections. Version 
one of the program was completed by January, 1993. 

Computer Program Use and Evaluation 

During the Spring 1993 semester (January - May), the computer program 
was used by the second professional year pharmacy students (76 total) in a 
required 1 credit hour "Introduction to Drug Literature" course. Although the 
School of Pharmacy faculty had approved an increase in credit hours of the 
course to accommodate the computer program, the change did not begin for this 
year's class. Since the computer program was equivalent in content length to at 
least a 1 credit hour class, the students were essentially asked to complete at least 
2 credit hours of coursework for 1 hour of credit. This affected students’ use of 
and opinions regarding the program (described in the “Evaluation/Project 
Results” section which follows). 

Each week, the students working in groups were assigned a different 
section of the program to complete on their own time. The subject matter was 
entirely self-contained in the computer program, i.e., class lecture time was not 
used to present the topics. Lectures were used for the other material traditionally 
taught in the course. Part of each hour lecture was available to answer any 
questions students had about their computer program assignment for that week. 
However, few questions about the computer material were usually raised by the 
students. To assist students in applying the information learned from the 
computer program to actual studies, each group of students (26 total groups of 
two to three students) received a different published drug efficacy study to 
review. Every other week during the course, the students were given questions 
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pertaining to their study that required an understanding of the corresponding 
material in the computer program. The investigators used the students' responses 
to identify areas in the program requiring clarification, revision, or expansion, 
and to make appropriate changes. Students were also asked to write down and' 
hand in their comments about each computer program section. The comments 
were reviewed and the program revised as needed. Some changes made in the 
program as a result of students' suggestions included the addition of "Previous" 
and "Next" buttons to each screen of the program. This allowed users to skip 

material already mastered without first having to complete all interactions on the 
screen. 

The program automatically recorded the amount of time that each 
student/group spent using the computer. The total amount of time the students 
actually spent using the computer program was compiled. Also, attitudinal 
surveys pertaining to each of the individual computer sections and a survey 
covering the program overall were completed by the students. 



The evaluation of the computer program used a pre-test/post-test design 
and contrast and study groups of students. During January 1992, a pre-test 
designed to determine baseline knowledge of drug literature evaluation principles 
was administered to the class of fourth year pharmacy students who represented 
the contrast (comparison) group in this project. These students did not receive 
drug literature evaluation material as part of their formal pharmacy education. 
This pre-test was very similar in content to the pre-test given to the study group 
of students (who used the finished computer program during year 2 of the 
project), and included questions pertaining to all aspects of drug efficacy study 
evaluation principles. The students were also asked questions about an actual 
drug efficacy study they were instructed to read in advance. A post-test, similar 
in content to that of the pre-test but longer in length, was administered to the 
contrast group students during their final class in May, 1992. Since students 
could not be randomly assigned into the contrast and study groups, demographic 
data were collected from each group to determine if they were comparable with 
regard to the following relevant attributes! age, grade point average, previous 
college degree (if any), subscription to journals, previous coursework in statistics 
or research methodology, prior exposure to the critical analysis of research 
studies, and previous participation in research projects. 

Although not part of the original project proposal, computer program 
evaluation data were also obtained from the pharmacy students in the expanded 2 
credit hour “Introduction to Drug Literature” course during the Spring 1994 
semester. A pre-test and post-test were administered to these students, and 
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computer use and altitudinal data were collected. The time spent using the 
computer program was also recorded for the Doctor of Pharmacy degree 
students who used the program during the Fall 1994 semester. 

Originally it was planned to administer a brief follow-up post-test to the 

rfschoof Th ,U f y ,, gr ° UP S ‘ Uden,S aftCr * he Clerkshi P Semester '"'he.r Anal year 
ot school. The follow-up post-test score in the study group was to be comnareri 

to the contrast students’ scores and used as a measure of the study students' P |ong- 
term knowledge retention. However, the only time this test could reasonably be 
administered was at the end of a comprehensive, several hours long post- 
clerkship examination, since the students are scattered throughout West Virginia 
during their clerkship semester. The post-test could only be very brief, given 
the length of the rest of the exam. As a result of this, combined with the fact that 
students knew it did not count toward their overall clerkship grades, the results 

were not felt to be meaningful and have not been used. 

Program Revisions and Updates 

The program has undergone considerable changes since the time it was first 
use in the Spring of 1993 to present. Shortly prior to the initial testing the 
program screens needed to be resized. The CBLC had purchased several new 
computers with small monitors; a program developed for a larger monitor using 
Authorware will disproportionately cut off the right and bottom portions of each 
screen when used on a smaller monitor. Thus, each screen of the program had to 
be re-formatted. Immediately following use by the first study group the 
investigators learned that the primary font used in the program, Chicago, was a 
poor font for instructional development. The investigators then selected a serif 

program 1 ” 0 l ° SUbstltUte and again had to chan 8 e each individual screen of the 

Minor changes in program content have been made continuously, based 
upon student feedback, reviewer’s comments, and investigator review 

SrwTT ° f tW ° e ?! irely neW SeCti ° nS also began f° llow ing computer use by 
the first study group. One was a “Correlation” subsection added to “Statistics.’’ 

The other, Program Review,” consists of a series of questions for each of the 
main sections (and subsections) of the computer program; a total of 15 sections of 
questions (with about 15-25 questions per section) were developed The program 
is designed to randomly choose about 10 questions each time the user reviews an 
individual section. The “Program Review’’ portion was completed prior to 
computer program use by the 1995 class. The investigators plan to continue to 
develop new questions for “Program Review,” so that a comprehensive question 



bank can be maintained. 
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From a instructional design perspective, it was also desired for each section 
or subsection of the computer program to be completed in an average of 30 
minutes or jess. Based on user data, several sections of the program were found 
to reqmre significantly more time to complete. The content of these sections 
(JoumaisTitles^^uthors Abstracts,” “Patients/Subjects,” “Controls, Design, 
Randomization, Blinding ) was further divided, and submenus were added that 
allowed for them to be completed in shorter time periods. Four 
sections/subsections still require over 30 minutes to complete (mean = 37-47 
minutes); plans are underway to similarly subdivide them. 

It was desired for the computer program to ultimately be available for use 
on two platforms, Macintosh and Windows. During conversion of the program 
from the Macintosh to a Windows format, the investigators initially used type 
ton ts for Windows that were the same as those used on the Macintosh (e.g. 
Palatino, New Century Schoolbook, Chicago, etc.). Unfortunately, it was later 
realized that when the Windows version is “packaged” by Authorware for 
distribution, the fonts are not automatically included. Since most Windows users 
do not have Macintosh compatible fonts, the investigators once again needed to 
change the fonts in the program. This time, the fonts were changed to 
accommodate those commonly used in Windows, e.g.. Times New Roman, Arial, 
Century Schoolbook). This revision is almost complete. 

New sections will continue to be developed for the “Evaluation of Clinical 
? x r . u ^ Studies program, in both platforms. These sections, “Regression” and 
Multivariate Analysis” for “Statistics” and a main “Discussion/Conclusions” 
portion, can be used by advanced level students (e.g., graduate students, Doctor 
of Pharmacy [Pharm.D.] students) and for riontraditional Pharm.D. program use 
Nontradihonal programs, to allow current B.S. degree practitioners to obtain a 
Fharm.D. degree on a part-time, external basis, have been developed by a few 

“i^Twm S far ’ a " d ‘ heir deve, ° pment is being planned by several 



Evaluation/Proiect Results 

_ f ? e ft*? 1 results ’ P° st - test results, and attitude survey findings from the 
contrast and study groups, and the computer use data from the study group are 

described in Appendix C -- “Development and Evaluation of a Computer- Assisted 
Instructional Program to Teach Critical Evaluation of Drug Studies.” A 
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summary of several of these findings follows. 
Pre-Test and Post-Test Scores 
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Both the mean and median pre-test scores for the 77 contrast group 
students who took: the exam were 21 out of 50 total points (42%), with a range of 
12.5 to 28 points out of 50 (25% to 56%). Surprisingly (since students knew that 
the pre-test did not count towards their overall course grade), the majority of 
students attempted to answer all the questions on the pre-test and appeared to have 
read the assigned published drug study in preparation for it. Since the contrast 
group received no instruction in evaluating clinical drug studies, it was 
anticipated that the post-test grades would not significantly increase (pre-test 
mean - 42%, post-test mean = 34%). However, the decrease in the mean post- 
test score compared to the pre-test was unanticipated, i.e., it was expected to at 
least remain the same. The primary reason for the post-test decline was thought 
to result from its administration during final exam week. Students were asked to 
review two published articles in preparation for the post-test; since they knew the 
test would not count towards their overall course grade (unlike their other 
finals), it was likely that the students did not thoroughly review the articles. To 
determine the extent to which this was true, both the pre-tests and post-tests were 
re-scored without the journal article questions. The revised pre- and post-test 

in fact similar (mean pre-test score = 43.8%, mean post-test score = 

42.2%). 



The mean pre-test score for the study group students was 44%, with a 
range of 29% - 56%. Although the difference between the contrast and study 
group pre-test scores achieved statistical significance (P=0.04), the two 
percentage point difference was not believed to be meaningful.’ The mean post- 
test score for the study group was 77%, with a range of 51% - 97%. This 
increase was statistically significantly different from both the pre-test score, as 
well as the contrast group post-test score. The results demonstrated that the 
computer program was associated with significant learning of the drug study 
evaluation material by students. This was verified by student attitudes, in which 
only 3% of students felt that they already knew most of the information in the 
program and 85% felt that they learned new information. 

Students in the Spring 1994 course received an average grade of 42% on 
the pre-test (median = 43%, range = 24% - 65%), compared to an average of 
83% on the post-test (median = 85%, range = 58% - 98%). The increase in post- 
test scores is significant and reinforces the program’s effectiveness in increasing 
students drug literature evaluation skills. Similar to the study group students. 
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only 4% of the Spring 1994 group felt they already knew most of the information 
in the program, and 84% believed they learned new information. 

Computer Use Time and Attitudes - Study Group 

a 9 st , udy . gro “P students spent a mean of approximately 1 1 hours (range = 
\ " 25 b ° urs) u ? mg the computer program. The attitudes of the study group 
students toward use of the computer program are shown in Table VI of Appendix 
. The primary complaint with regard to the program was the amount of time 
required to complete it with the second most frequent complaint being the lack 

rnmn?f°f tS f° r 3 w ° rkb ° ok to use m conjunction with the program. Another 
complaint of several students related to a lack of appreciation of the need to know 
how to critically analyze and interpret published drug studies. Unfortunately 
this problem is unlikely to be resolved until changes are accomplished in the 
entire curriculum. The study group students had limited pharmacy practice 
experience at the time they took the “Introduction to Drug Literature” course 
and the curriculum did not require them to retrieve and review published drug 
studies until their final year (and even then, to only a limited extent). However 
this situation should be remedied by two major curricular changes one in 
progress and the other planned. First, problem-based learning (PBL) is being 
unplemented in our pharmacy curriculum as a result of a current FIPSE grant. 

I his is currently affecting our first professional year pharmacy students. PBL 
requires greater use of supplemental information resources (beyond course 
ecture notes). Second, plans are underway for conversion to a six year entry- 
level pharmacy curriculum, with significantly greater emphasis on clinical 
expertise and practice-based experience. The ability to critically evaluate the 
literature will be readily apparent to students in such a curriculum. 

Computer Us e Time and Attitudes - 1994 Data 

,, „ Phar ™ c >' students 'n I** “Introduction to Drug Literature” course during 
e Spring 1994 semester spent 11.1 + 5.1 hours (mean + SD) using the computer 
program (median = 10.4 hours, range = 1.2 to 26.1 houre). The Doctor of 
Pharmacy degree students (six total) in the Fall 1994 semester spent an average of 
11.5 hours using the program (range = 8.4 to 17.8 hours). The mean times are 
similar to those obtained from the study group students; thus, the program length 
has been fairly accurately determined. ® 

AnnenH^v A tti TU in f al reSUltS fr ° m ,h<! S P rin 8 1994 StudentS ™ shown in 

Appendix D. The findings are comparable to the previous year, with the most 

common complaint still being the amount of time required. However, the 
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computer program workbook, a new addition, was considered to be helpful by 
63% of respondents. Several students expressed a desire to simply receive 
handouts and attend lectures rather use the computer program. This is not 
surprising considering that the vast majority of the pharmacy curriculum 
consists of lectures and exams which test rote memorization of material. Also 
expansions scheduled for the CBLC, used by the students to view the computer 
program, have been delayed. Thus, it was often crowded and some students 
indicated difficulty in working there. 

Success of Project 



Based upon the pre-test and post-test comparisons, as well as student 
opinion, the program and project as a whole have been deemed to be a success 
further, in 1993 the investigators submitted the computer program and a 
portfolio of project-related work to AACP for consideration to receive one of 
their Innovations in Teaching” awards. Our project was one of the three 
selected as winners. As a winner, $1200 was awarded to travel to San Diego to 
present (in both a poster and platform format) the results from this project The 
program was demonstrated later that year at the American Association of Medical 
Colleges annual meeting as well as the SCAMC (Symposium on Computer 
Applications in Medical Care) meeting. As a result of these presentations, and 
others within West Virginia, individuals from at least 30 different institutions 
(including three pharmaceutical manufacturers) have expressed interest in 
receiving a copy of the program. 

Plans for Continuation and Dissemination 

The computer program will continue to be used in the WVU School of 
“ C >; s curriculum and will also be incorporated into the School of 
Medicine’s curriculum. Evaluative data will continue to be obtained from 
students using the program to assist with future revision efforts. 

To prepare for distribution of the “Evaluation of Clinical Drug Studies” 
program, copyright registration was obtained in August, 1994. Rather than 
simply distributing the program to those who expressed an interest in it the 
investigators have been working on a distribution mechanism by which the costs 

time ’ maiIm g, etc.) can not only be recovered, but a small amount of 
additional money obtained to allow for continued program revision. The 
investigators are also interested in reaching a variety of different audiences with 
the program: students in several health professions, pharmacy and medicine 
residents, educators, health care practitioners, journal editors and reviewers, and 
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pharmaceutical industry personnel. Since the target audience is quite broad, the 
investigators are considering working with a pharmaceutical manufacturer in its 
program distribution efforts. 

An educational liaison for the pharmaceutical manufacturer Astra/Merck 
viewed the computer program a few months ago and, as a result, provided the 
investigators with a $5000 unrestricted educational grant to continue work on the 
project. The investigators are using this money to begin to develop a condensed, 
practitioner s version of the computer program. This version will focus on the 
most important questions to ask when reviewing a published drug study and will 
be able to be completed within a shorter time period. It is envisioned that the 
program could be used in a stand-alone manner or as a supplement to the detailed 
version. In March 1995, the educational liaison and three other individuals from 
Astra/Merck headquarters visited WVU for the purpose of discussing areas of 
mutual interest involving the computer program. The meeting was productive 
and several future directions for program development were considered. 
Astra/Merck should be contacting the investigators shortly with a specific 
proposal for collaboration. If they do not wish to become involved with 
distributing the current program, the investigators are pursuing a working 
arrangement with the Health Sciences Consortium for program distribution to 
educational institutions. 

A colleague in the WVU School of Pharmacy is interested in expanding the 
material presented in the Evaluation of Clinical Drug Studies” program to 
include the critical evaluation of pharmacokinetics studies. In this regard, a brief 
proposal was submitted for funding to the Drug Information Association (DIA) 
for the development of an “Evaluation of Pharmacokinetic Drug Studies” 
program. This program, when completed, could be used in a stand-alone 
manner or in conjunction with the current program. Preliminary interest in 
funding the proposal has been expressed by the DIA, and its president. Dr. Louis 
A. Morris of the Food and Drug Administration (FDA), has asked to receive a 
copy of the current “Evaluation of Clinical Drug Studies” program to review. 

Dr. Morris has asked for the IBM Windows version, which should be completed 
and sent within the next two weeks. If Dr. Morris is favorably impressed by the 
program, the opportunities for its use within government agencies and 
throughout pharmaceutical industry should be enhanced. Also, if the proposal is 
funded by the DIA, they have already requested that both the current program 
and the developed “Evaluation of Pharmacokinetic Drug Studies” program be 
demonstrated at the DIA Annual Meeting in June, 1996. 



The principal investigator also intends to gain greater proficiency with the 
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use of Authorware Professional® in order to facilitate her work with the 
computer program. A new version of Authorware Professional® (Macintosh and 
in ows is ue to be released in April 1995. The principal investigator plans to 
attend a three day Authorware advanced training workshop in June 1995 as part 
oi her development efforts in this regard. 



Summary and Conclusions 

The investigators have truly enjoyed their involvement with this project. A 
comprehensive computer instructional program has been developed which can 
successfully educate students to critically analyze published drug efficacy studies, 
further, in the process of completing the project, the investigators have learned a 
great deal more about instructional design, computer assisted instruction and the 
implementation of new educational strategies. The principal investigator in 
particular has also become relatively proficient in the use of a powerful software 
development program, Authorware Professional®. As a result of this 
undertaking, the investigators can envision other types of computer instructional 
projects that would be of value to develop, especially as curriculum revisions are 

planned both in the Schools of Pharmacy and Medicine at WVU, and throughout 
the nation. & 

Before initiating a project such as this, individuals need to be aware of the 
considerable development time and effort required. This time and effort are not 
only required for the initial computer program work (including content 
preparation, formatting, programming, and testing), but for revision 
maintenance and distribution as well. Working with different computer 
platforms and equipment (similar to those which a variety of users might possess) 
produces unique challenges that must be anticipated and overcome. Program 
records and documentation are also important to maintain throughout the project, 
finally, the topic(s) for which computer instructional materials are developed 
should be well-suited for this media. The satisfaction of having developed a 

program that one is proud of, however, can compensate for the extraordinary 
eiiort expended for it. 
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Date 


Appendix B 
Timetable for Project 

Year 1 

Activity 


September, 1991 


Authorware training 


September, 1991 - August, 1992 


Database development, field testing 


January - May, 1992 


Pre-test and post-test administered to contrast 
group of pharmacy students (in their 4th year of 
the curriculum) 


Date 


Year2 

Activity 


September, 1992 


Database development (continued) 


September, 1992 - May, 1993 


Collect data concerning knowledge of drug 
literature evaluation techniques a second time from 
the contrast group of pharmacy students (in their 
5th year of the curriculum) 


October - December, 1992 


Pilot testing of database by selected students 


November - December, 1992 


Revision and refinement of database 


January - May, 1993 


Pre-test and post-test administered and database 
evaluated using study group of pharmacy students 
(in their 4th year of the curriculum) 


June - August, 1993 


Data analysis and interpretation 


Date 


Year 3 

Activity 


September, 1993 - May, 1994 


Collect data concerning knowledge of drug 
literature evaluation techniques a second time from 
the study group of pharmacy students (now in 
their 5th year of the curriculum) 


June - August, 1994 


Final data analysis and interpretation 
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Appendix C 



Development and Evaluation of a Computer* Assisted Instructional 
Program to Teach Critical Evaluation of Drug Studies 1 * 2 

Marie A. Abate and Arthur I. Jacknowitz 

School of Pharmacy, West Virginia University, Morgantown WV 26506-9520 

James M. Shumway 

School of Medicine, West Virginia University, Morgantown WV 26506 

Anne H. Nardi 

College of Human Resources and Education, West Virginia University, Morgantown WV 26506 



INTRODUCTION 

Computer-assisted instruction (CAI) has been used in medi- 
cal education for several years, although its development 

1 Supported by grant PI 16B10076 from the U.S. Department of Education, 
Fund for the Improvement of Postsecondary Education (FIPSE). 

2 Presented by Marie Abate in part during the Innovations in Teaching 
Award poster and platform sessions, AACP Annual Meeting, July 1993, 
San Diego, CA. 



continues to exist in an evolutionary stage(l ,2). C AI has also 
been incorporated into various subject areas in the phar- 
macy curriculum(3-7), but to a relatively limited extent. 
Educators in both pharmacy and medicine have been en- 
couraged to continue to utilize and exploit the full potential 
of computers in the curriculum(l,8). 

The education of students in the techniques of critical 
appraisal of drug studies has been identified as a deficiency 
in many health sciences curricula(9,10). Errors in research 
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design and inconsistencies in the reporting of study results 
exist in professional pharmacy and medical journals, despite 
the presence of editorial boards and the peer-review pro- 
cess(ll,12). Therefore, thorough and accurate interpreta- 
tion of the primary literature is essential for assuring that 
patients receive appropriate drug therapy. Health care edu- 
cators have reported that students must know how to criti- 
cally analyze drug studies and understand statistical meth- 
ods in order to interpret such studies(ll,13,14). A recent 
AACP background paper has identified the ability to ap- 
praise and evaluate clinical drug studies as a skill required of 
entry-level pharmacy practitioners in order to provide phar- 
maceutical care(15). 

Several checklists, scoring systems, and algorithms have 
been published to assist health care providers in evaluating 
clinical studies(16-18). It was anticipated that a CAI pro- 
gram would also be a useful method for teaching students 
how to critique published drug studies, since numerous 
examples and levels of explanation could be incorporated 
into a computer and accessed and reviewed by users to the 
extent desired. In addition, a program of this type would 
promote active learning by students. The objective of our 
project was to develop and evaluate a comprehensive CAI 
program which would educate students to evaluate all as- 
pects of published drug efficacy studies thoroughly and 
critically. 

METHODS 

Program Development. Authorware Professional® 
(Macromedia, San Francisco, CA) was used for develop- 
ment of the computer instructional program, initially on the 
Macintosh platform. This software allows for the develop- 
ment of instructional material which incorporates sound, 
graphics, and animation, without the need for programming 
expertise by the developer. In addition, it allows for conver- 
sion of the program to a PC Windows environment. An 
Authorware® training program which provides an overview 
of the basic operation of the software was offered by the 
manufacturer and attended by the investigators. 

Development of the instructional program began in 
September, 1991. An outline of topics to be covered in the 
program with corresponding learning objectives (Appendix 
A) was prepared. The goal was to develop a self-contained, 
stand alone, interactive CAI program that would educate 
students to critically analyze and evaluate all aspects of 
published drug studies, with a focus on drug efficacy. For 
example, in the area of statistics, students would be expected 
to understand statistical concepts, not to actually compute 
statistics tests. Since the material in the program had not 
been taught previously, the content first needed to be out- 
lined and text prepared. The CAI program was intended to 
replace lectures and included instruction, examples, ques- 
tions, responses, and feedback, as well as sound, graphics, 
plus the ability for students to receive remediation or con- 
tinue to progress through the program. Content develop- 
ment and formatting of the material into an interactive 
design continued from September, 1991 until September, 
1992. Initial refinement of the program by the investigators 
prior to its formal evaluation occurred from September to 
December, 1992. A senior pharmacy student also reviewed 
the program in detail during this period, prepared written 
comments/suggestions related to its clarity and format, and 
recorded the time required for the completion of each 
computer section. 

ErJcIST COPY AVAlLABfc& lc * nJourna ' o,Pharm 



Program Evaluation. The evaluation plan utilized a pretest/ 
posttest design with control and test groups of pharmacy 
students. The control or contrast group consisted of the 77 
second professional year pharmacy (BS degree) students 
enrolled in a required one credit hour Introduction to Drug 
Literature (IDL) course in the spring semester of 1992. This 
group received no instruction in critical drug study evalua- 
tion techniques, since undergraduate pharmacy students at 
West Virginia University had not received required formal 
instruction in this area prior to development of the com- 
puter program. Therefore, the control group served to 
measure the exposure of students to any drug study evalua- 
tion concepts received during other courses. The control 
group was administered a pretest, designed to determine 
their understanding of important characteristics of clinical 
drug studies and their ability to analyze and evaluate such 
studies, at the beginning of the IDL course. The pretest 
consisted of multiple choice and short essay type questions, 
part of which pertained to an actual published drug efficacy 
study. Cognitive skills and not simply factual recall were 
tested by many of the questions. Examples of the pretest and 
posttest questions are provided in Appendix B. A posttest 
similar in content and design to the pretest, with the excep- 
tion of additional questions pertaining to two published 
efficacy studies, was administered at the end of the course. 

The test group consisted of the 76 second professional 
year pharmacy students enrolled in the IDL course during 
the spring semester of 1993. Demographic and other perti- 
nent background data were obtained from both the control 
and test groups to determine if they differed significantly 
with regard to important attributes. At the start of the IDL 
course, the test group was administered a pretest almost 
identical to that completed by the control group the previ- 
ous year. The computer program was designed to be very 
user friendly and a simple knowledge of Macintosh basics, 
e.g., clicking and dragging, was sufficient for its use. The 
class was divided into smaller groups and given 30 minutes 
of hands-on introduction to the use of the Macintosh com- 
puter and how to access the drug study evaluation program. 
Basic Macintosh operation tutorials were also available on 
the computers for students to review on their own, if desired. 
The program was installed on 14 Macintosh Ilci or si com- 
puters in the Health Sciences Center’s Computer Based 
Learning Center (CBLQ for student use throughout the 
semester at any time the lab was open, weekdays or week- 
ends. 

The students were then divided into groups of two or 
three each and assigned to review one to two sections/ 
subsections of the program every two weeks during the 
semester. However, students were allowed to review the 
program individually if computer space was available. Due 
to an inability at that time to increase the credit hours of the 
IDL course to accommodate the greater workload, the 
computer assignments represented additional effort for the 
class; this was explained to students at the beginning of the 
course. Students were given a copy of the learning objectives 
and a topical outline to use while viewing the program. Each 
student group was also given a different published clinical 
drug efficacy study to review. Six exercises assigned through- 
out the semester asked questions which related to the clini- 
cal study and which evaluated students’ comprehension of 
the instructional program’s contents. In addition, the exer- 
cises alerted the investigators to content areas in the com- 
puter program which might have been incomplete or un- 
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Table I. Program content outline 

Section 1 - Clinical Literature and Types of Studies 
Section 2 • Introduction/Background 
Section 3 - Journals, Titles, Authors, Abstracts 
Section 4- Patients/Subjects 

Section S - Controls, Design, Randomization, Blinding 
Section 6 - Treatment Considerations 
Drug Considerations 
Study Settings 
Patient Factors 
Section 7 • Measurements 
Section 8- Statistics 

Variables, Data, and Distributions 
Measures of Central Tendency 
Measures of Variability 
Statistical Inference 
Parametric Tests 
Nonparametric Tests 
Section 9 - Data Handling 



clear. Students were asked to maintain a “log” of problems/ 
comments/suggestions that arose while reviewing the pro- 
gram. 

Surveys designed to determine student attitudes toward 
the computer program overall and toward its different 
sections were administered four times during the semester. 
The surveys consisted of a series of statements which stu- 
dents were asked to rank using a 5 point Likert-type scale 
( Le ., 5 = strongly agree, 1 = strongly disagree). The final 
survey also asked students to expand upon those statements 
which they rated positively or negatively. The computer 
automatically recorded the amount of time which the stu- 
dents/groups spent using the program and the total com- 
puter time was compiled for each student. The program 
allowed the investigators to determine whether or not an 
individual section or subsection of the program was ac- 
cessed, although it was not possible to obtain an accurate 
time breakdown for each section due to the program’s 
design, Le., the freedom of students to move from one 
section to another during a session. 

A posttest containing questions almost identical to those 
given to the control group was administered to the test group 
students at the end of the semester. Student f-tests or 
ANOVA were performed to compare the pretest and posttest 
scores between the control and test groups. Paired r-tests 
were used to determine the differences between the pretest 
and posttest scores within the control and test groups. An 
alpha level of 0.05 was considered the cut-off for determin- 
ing statistical significance. Cronbach’s coefficient alpha 
method was used to evaluate the internal reliability of the 
attitudinal survey items(19). 

RESULTS 

Program Preparation, Content and Use. The instructional 
program developed, entitled “Evaluation of Oinical Drug 
Studies,” consisted of nine main sections, two of which 
contained several subsections. A total of 16 separate sec- 
tions and subsections were prepared. Since the computer 
material was developed to stand alone, only 15 to 20 minutes 
per week of classroom lecture time were allotted for pro- 
gram discussion, primarily to answer any content-related 
questions from students. The content outline of the program 

O i 



is shown in Table I. The entire program utilized approxi- 
mately 6 MB of disk space and print-outs of the computer 
screens have filled four large ring binders. Although stu- 
dents were divided into small groups to facilitate access to 
the CBLC computers, slightly more than half of the students 
(53 percent) indicated that they preferred to work alone 
instead of in a group. The majority of students (87 percent) 
felt they received sufficient initial instruction as to how to 
access and use the program. 

It was difficult to determine the total amount of prepa- 
ration time required for the initial version of the instruc- 
tional program, given the number of investigators involved 
in its development, the often part-time nature of their work 
on the program, and the time spent reading, reviewing, and 
revising each prepared section. Although Authorware® was 
designed to be user friendly, there was still a considerable 
learning curve which needed to be overcome in order to use 
it optimally. In addition, a problem arose midway through 
the development of the program when the CBLC bought 
additional Macintosh computers with a smaller monitor size 
than those already present in the lab. To allow the program 
to run on the smaller monitor, each screen of information 
had to be re-sized individually, which added several hours to 
the preparation time and resulted in two versions of the 
program (Le., one small screen and one full screen) being 
maintained and operated at the same time. The investiga- 
tors felt that approximately 15 hours would be necessary in 
order for users to cover thoroughly all the material and 
examples presented in the computer program. All totaled, it 
is estimated that up to 50 hours were spent for the program’s 
preparation for each hour of material developed. 

The students spent a total of 10.9 ± 4.4 hours (mean ± 
SD) reviewing the instructional program (minimum time = 
4.0 hours, maximum time = 25.0 hours, median time = 9.9 
hours). However, as shown in Table II, a total of 49 students 
(64 percent) skipped at least one section/subsection of the 
program, with 32 percent skipping three or more sections/ 
subsections and nine percent missing over half of the pro- 
gram. This was consistent with the primary complaint of the 
students with regard to the program, Le. , the amount of time 
required for only a one credit hour course (see “Attitude 
Assessment”). Two students specifically stated that they did 
not have the time to spend on the program given the rest of 
their workload and simply stopped reviewing the sections. 

Pretest and Posttest Scores. Relevant background data 
pertaining to the control and test groups of students are 
shown in Table III. The control and test groups had virtually 
identical ages and mean professional program GP As. Slightly 
more control students had prior experience with statistics or 
research methods than those in the test group. This might 
have been expected to result in better performance on the 
pretest by those in the control group, although this did not 
occur. 

The pretest and posttest scores from both the control 
and test group students are shown in Table IV. The differ- 
ence between the control and test group pretest means 
barely achieved statistical significance, with a two percent- 
age point difference. However, a significant increase in the 
posttest compared to the pretest scores occurred only in the 
test group students, demonstrating that the computer pro- 
gram significantly improved the students’ knowledge of 
critical drug study evaluation techniques and their ability to 
evaluate primary literature. 
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Table II. Sections/subsections skipped 



Number of sections/ 


Number (percent) 


subsections skipped 


of students 


1-2 


25 (32.9) 


3-5 


13 (17.1) 


6-8 


4 (5.3) 


9-11 


7 (9.2) 



Table III. Backgrou rid data from control and test groups 



Characteristic 


Group 

Control 


Test 


Males 


18 


23 


Females 


59 


53 


GPA a 


3.28 ± 036 


3.29 ±038 


Age (yrs) 


23 ± 3 


22 ± 3 


Percent with previous 
stat. course 


18.7 


143 


Percent with previous research 
methods course 


12.0 


53 


Percent with prior degree 


12.0 


10.7 



‘Upon entrance into IDL course. 



much time being required to complete the program and the 
lack of a detailed handout/workbook to use. Students were 
informed the first day of the IDL course about the extra 
work expected from them in order to complete the com- 
puter program assignments, beyond that normally associ- 
ated with the course. Several students specifically com- 
mented on their attitudinal survey that they did not receive 
enough credit for the work in the course. Related to the time 
requirement, the second most frequent complaint from 
students was that they had to take too many notes while 
reviewing the computer program and many asked to receive 
accompanying handout material or a workbook. 

Several students provided written comments other than 
those pertaining to the time commitment and handouts. 
Some students felt that pharmacists did not need to know 
how to evaluate journal articles, as illustrated by the follow- 
ing statements: “This class is Drug Literature — not evaluat- 
ing drug studies and it is my belief that this type of knowl- 
edge is not useful to the common BS RPh,” “In the ‘typical’ 
pharmacy in community practice you will not be reading 
study articles. You will read things like American Pharmacy 
[sic] or Drug Topics ...Ido feel this course is important for 
physicians,” and “This class should be an elective for some- 
one who wants to go into research.” Other students com- 



Table IV. Pretest and posttest percent grades 



Student group 


Pretest 
mean ± SD 


Range 


Posttest 
mean ± SD 


Range 


Control 


41.6±3.2* ,c 


26-56 


33.9 ±7.0= 


15-50 


Test 


43.6 ± 3.1 »- b 


29-56 


77.4 ± 9.3 b 


51-97 



*P=0.04; b / > =0.0001; c f»=0.0001. 



Table V. Computer time spent by students receiving 
lowest and highest posttest grades 





Time (hrs) using program 


Posttest grade (N) 


Mean Range 


50 - 59% (5) 


6.9 6.0 - 7.9 


90 - 99% ( 6) 


17.4 10.6 - 25.0 



An analysis of the total time spent by students reviewing 
the computer program vs. the posttest scores found a rela- 
tively weak correlation overall between these two variables 
(r=0.44). However, a considerable difference existed be- 
tween the time spent using the computer program by the 
students with the lowest vs. the highest scores on theposttest, 
as shown in Table V. The students who received a score of 
90 percent or better on the posttest spent on average slightly 
over 10 hours more time reviewing the computer program 
than those with the lowest scores. As expected, the students 
with a posttest score of 90 percent or better skipped fewer 
(mean = 1; 0 to 3) sections/subsections of the program, 
compared to those with the lowest scores (mean = 5; 1 to 10 
sections/subsections skipped). 

Attitude Assessment. A coefficient alpha reliability mea- 
sure of 0.93 was calculated for the attitudinal survey items. 
A summary of the final attitudinal survey which assessed the 
students’ overall opinions of the computer program is shown 
in Table VI. The major negative comments involved too 



mented: “I like learning on my own!!,” “Only improvement 
is to make it available for home use,” and “I enjoyed the 
computer learning type of setup.” 

Students were also asked whether they would like to see 
the use of more, less, or the same amount of various com- 
puter program features such as sound, moving objects, or 
narration. These features had been incorporated into the 
program to varying extents. The students indicated they 
would prefer less answers that needed to be typed in (44 
percent of respondents) and less need to drag responses on 
the screen (55 percent), but more animation of objects (41 
percent), more audio narration (41 percent), and more 
sound effects (47 percent). 

DISCUSSION AND CONCLUSIONS 

Educators have clearly stated that need exists for pharmacy 
students and practitioners to evaluate clinical studies criti- 
cally and to understand and analyze research design(15,20). 
A combination of factors led the investigators to develop a 
CAI program which would teach students these skills: the 
desire to enhance the students’ ability to think critically and 
solve problems; the desire to develop course materials which 
promote active student learning; the need topromote greater 
flexibility in the curriculum and to develop programs which 
could be used at remote off-campus sites; the desire to 
develop instructional me thodologieswhichwould ultimately 
release faculty time for other activities; and the lack of 
availability of any existing CAI programs in this area. Sev- 
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Table VI. Overall attHudlnal survey results 



Frequencies (Percent) 

Strongly Disagree to 



Statement 


Mean 


agree to 
agree (5,4) 


Neutral (3) 


strongly 
disagree (2,1) 


1. Already knew most of information in program 


1.9 


2.7 


5.4 


91.9 


2. Learned new information from program 


4.1 


85.1 


93 


5.4 


3. Learned useful and relevant information 


3.4 


55.4 


25.7 


18.9 


from computer program 


4. Pharmacists need to know how to evaluate and 


3.6 


64.8 


16.2 


19.0 


interpret journal articles 


5. Liked the way screens designed and information 


3.7 


68.9 


18.9 


12.2 


presented 


6. Liked colors used 


4.0 


83.8 


9.4 


6.8 


7. Difficult to understand the material presented 


2.7 


24.3 


23.0 


52.7 


in program 


8. Information in program geared to appropriate 


3.5 


65.8 


17.8 


16.4 


level 


9. Using computer program was interesting 


3.2 


51.4 


20.8 


27.8 


10. Took too long to complete sections* 


4.1 


81.0 


9.0 


10.0 


1 1 . Prefer to work alone on computer 


3.3 


533 


11.3 


35.2 


12. Would prefer to use computer to learn 


3.1 


41.7 


19.4 


38.9 


material than attend lectures 


13, Overall, quality of program was high 


3.5 


62.2 


20.2 


17.6 


14. A major deficiency of program was lack of 


4.1 


79.7 


6.8 


133 


detailed handout/workbook 


15. Programs of this type are valuable 


33 


533 


11.3 


353 


educational tools 



* Combined results from first three altitudinal surveys. 



eral reports in the pharmacy literature have shown the 
effectiveness of CAI in various subject areas, generally 
when compared to traditional lectures(3,6). However, a 
recent review of CAI research concluded that enhanced 
effectiveness and efficiency of this modality relative to other 
forms of instruction has not been established(2). 

In our study, since the control group received no in- 
struction in drug literature evaluation techniques, it was not 
possible to compare the program’s effectiveness to a tradi- 
tional lecture format. However, the “Evaluation of Clinical 
Drug Studies” program was shown to significantly increase 
students’ drug efficacy study evaluation knowledge and 
skills. Since the time requirement for the IDL course with 
the computer component was excessive for one credit hour, 
a significant number of students skipped sections of the 
program. It is anticipated that student posttest scores and 
therefore knowledge of the subject will further improve as 
a result of curricular changes to be implemented at the 
school of pharmacy during the spring semester of 1994; Le., 
an increase in credit hours of the IDL course from one to two 
to accommodate the computer program, together with less 
required credit hours overall that semester. 

In the control group students, the mean posttest score 
actually decreased in comparison to the pretest score, which 
was unanticipated, Le . , it was expected to remain stable. The 
primary reason for the posttest score decline was thought to 
result from its administration during final exam week. The 
students were asked to read and review two published 
articles in preparation for the posttest. Since they knew the 
test would not count towards their overall course grade 
(unlike their other finals), it was highly probable that the 



students did not thoroughly review their articles. To deter- 
mine the extent to which this was true, both the pretests and 
posttests were re-scored without the journal article related 
questions. The revised pre- and posttest scores were in fact 
similar (mean pretest score = 43.8 percent, mean posttest 
score = 42.2 percent). 

Despite the complaints from several students that they 
did not receive enough credit for their work in the IDL 
course, students generally agreed that they liked the 
program’s screen design and way in which the information 
was presented. They reported that they learned new infor- 
mation from the program, the information was geared to the 
appropriate level, and the overall quality was high. The 
students were almost equally divided as to whether they 
preferred to use the computer to learn the material or attend 
lectures. Other studies have shown that while pharmacy 
students would like to see CAI used for portions of courses, 
the majority did not desire an entire course to be given via 
computer, or for the computer to completely replace lec- 
tures^). With an increase in credit hours of the IDL 
course beginning in 1994, the intention is to allow students 
one hour per week of class time for use of the computer 
program. This should provide students with adequate time 
to complete all sections of the computer program, which 
should then translate into an even greater improvement in 
posttest scores. Portions of the remaining classes will be 
used for small group discussions of published efficacy stud- 
ies and to review the course exercises pertaining to drug 
studies. In this way, material covered in the computer pro- 
gram can be reinforced by actual contact time during class. 

Future plans for the “Evaluation of Clinical Drug Stud- 
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ies” program include the addition of regression analysis 
information and a section describing the analysis of the 
Conclusion/Discussion portion of a published study, devel- 
opment of a DOS Windows version, the incorporation of 
greater amounts of sound and animation, the addition of 
more examples for students to review as desired, and the 
preparation of an accompanying workbook. Version 2.0 of 
the program will be distributed to interested schools and 
colleges in late 1993. It is also planned for the program to be 
used in the postbaccalaureate doctor of pharmacy degree 
curriculum, the medical school curriculum, by practitioners 
for continuing education, and at off-campus sites, and for 
sections of the program to be used in other courses in the 
pharmacy curriculum. It is important that comprehensive, 
time intensive CAI programs be adaptable for use in mul- 
tiple ways. This will assure that the development and main- 
tenance costs of such endeavors will be appropriately com- 
pensated. 

In summary, the “Evaluation of Clinical Drug Studies” 
program was a unique and successful addition to the phar- 
macy school curriculum. The development and evaluation 
of similar programs in different content areas should con- 
tinue to be explored. 

Acknowledgement The authors gratefully acknowledge 
the programming and formatting work of Kelly Pryor for 
development of the Evaluation of Clinical Drug Studies 
program. 

Amer. J. Pharm Educ. , 57, 416-424(1993); received 9/29/93. 

References 

(1) Piemme, T.E., “Computer-assisted learning and evaluation in medi- 
cine, 1 "JAMA, 26$, 367-372(19 88). 

(2) Keane, D.R., Norman, G.R., and Vickers, J., “The inadequacy of 
recent research on computer-assisted instruction," Acad Med , 66, 
444-448(1991). 

(3) Newton, G.D., Popovich, N.G., and Lehman, J.D., “Development 
and evaluation of computer-assisted guided design for problem solv- 
ing instruction in self-care pharmacy practice," Am. J. Pharm. Educ., 
55, 301-310(1991). 

(4) Anderson-Harper, H.M., Mason, H.L., and Popovich, N.G., “Devel- 
oping, implementing, and evaluating microcomputer instruction in a 
nonprescription drug course," ibid, 52, 259-266(1988). 

(5) Narducci, W.A., “Development, implementation, and cost-assess- 
ment of an integrated computer-assisted instruction course on drug 
interactions," ibid , 49, 145-149(1985). 

(6) Clem, J.R. , Murry, D.J., Perry, P.J., and Alexander, B., “Performance 
in a clinical pharmacy clerkship: computer-aided instruction versus 
traditional lectures," ibid, 56, 259-263(1992). 

(7) Hurst, A.K., Noguchi, J.K., and Besinique, K.M., “Use of computer- 
assisted case presentations in a clinical therapeutics course," ibid , 52, 
56-58(1988). 

(8) Anderson-Harper, H.M., Mason, H.L., and Popovich, N.G., “Atti- 
tudes and beliefs of pharmacy students about using computers for 
instruction," ibid , 54, 263-268(1990). 

(9) Nelson, Jr., A. A., “Research methods for pharmaceutical practice," 
Am. J. Hosp. Pharm. , 37, 107-110(1980). 

(10) Springer, J.R., and Baer, LJ., “Instruction in research -related topics 
in U.S. and Canadian medical schools,” J. Med Educ., 63, 591- 
595(1988). 

(11) Haynes, R.B., “Loose connections between peer-reviewed clinical 
journals and clinical practice,” Ann. Intern. Med, 113,724-728(1990). 

(12) Ilersich, A.L., Arlen, R.R., Ozoiins, T.R.S., Einarson, T.R., Mann, 
J.L., Segal, H.J., “Quality of reporting in clinical pharmacy research,” 
Am. J. Pharm. Educ. , 54, 126-131(1990). 

(13) Polk, R.E., and Hepler,C.D.,“ Controversies in antimicrobial therapy: 
critical analysis of clinical trials,” Am. J. Hosp. Pharm. , 43, 630- 
640(198 6). 

(14) Boyce, E.G., and Nappi, J.M., “Is there significance beyond the /- 
test," Drug IntelL Clin. Pharm., 22, 334-335(1988). 



(15) Commission to Implement Change in Pharmaceutical Education, 
“Background paper II. Entry level, curricular outcomes, curricular 
content and educational process,” AACP News (March Special Re- 
port), 1-10(1991). 

(16) Sackett, D.L., “How to read clinical journals: V: to distinguish useful 
from useless or even harmful therapy,” Can. Med Assoc. J. , 124, 1 156- 
1162(1981). 

(17) Krogh, C.L., “A checklist system for critical review of medical litera- 
ture,” Med Educ. , 19, 392-395(1985). 

(18) Gardner, M.J., Machin, D., and Campbell, M.J., “Use of checklists in 
assessing the statistical content of medical studies,” in Statistics with 
Confidence (edits. Gardner, M.J., and Altman, D.G.), British Medical 
Journal, London, England (1989) pp. 101-108. 

(19) Mehrens, W.A., and Lehmann, I.J., Measurement and Evaluation in 
Education and Psychology, 2nd ed., Holt, Rinehart and Winston, New 
York NY (1975) pp. 97-99. 

(20) Draugalis, J.R., and Slack, M.K., “A course model for teaching 
research evaluation in colleges of pharmacy,” Am J. Pharm Educ. , 
56, 48- 52(1992). 



APPENDIX A. LEARNING OBJECTIVES 

As a result of completing the computer program, the participant 

should be able to: 

L Section 1 — Clinical Literature and Types of Studies 

a. Know the two basic types of medical research studies. 

b. Describe the format of the four observational studies 
(descriptive [case-series], case-control, cross-sectional, 
cohort) and recognize their advantages and disadvan- 
tages. 

c. Describe experimental studies and briefly discuss their 
advantages and disadvantages. 

2. Section 2 — Journals Titles Authors Abstracts 

a. Recognize what the goals of a journal should include. 

b. Describe what a journal citation is and its significance. 

c. Describe the reason that advertising is included in a 
journal and its effect, if any, on quality. 

d. Discuss the purpose and functions of an editorial board 
and its members. 

e. Describe the peer review process and its purpose. 

f. Discuss the potential conflicts of interest for authors of a 
published study. 

g. Describe the purpose of the abstract of a published 
article. 

3* Section 3 — Introduction/Background 

a. List the four points which the Introduction/Background 
section of a published study should include. 

b. Describe the two types of hypotheses. 

c. Identify whether an alternative hypothesis is one-tailed 
(sided) or two-tailed (sided). 

d. Restate (as necessary) the objective of a study in the form 
of a null hypothesis. 

e. Determine whether or not a study's objective is appropri- 
ately stated. 

4. Section 4 — Patients/Subjects 

a. Describe what the inclusion criteria and the exclusion 
criteria refer to in a study. 

b. Discuss the importance of the inclusion and exclusion 
criteria in regard to a study's results and their interpreta- 
tion. 

c. Describe how a homogeneous or heterogeneous sample 
can affect the interpretation of a study's results. 

d. Briefly describe the types of random sampling and indi- 
cate why it is preferred over nonrandom sampling. 

e. Describe selection bias and how it could be minimized in 
a study. 

f. Describe the meaning of “informed consent" and its 
importance. 
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*g. Indicate when the investigators should determine the 
number of subjects which they should include in their 
study. 

5. Section 5 — Controls, Design, Randomization, Blinding 

a. Explain why controls are important to utilize in a study. 

b. Describe the types of controls which exist. 

c. Indicate the type(s) of controls which would allow one to 
truly determine whether a study drug itself has any 
efficacy. 

d. Describe the concurrent control (parallel treatment), 
“before and after” (time series), and cross-over study 
designs. 

e. Recognize the advantages and disadvantages of each type 
of study design. 

f. Describe the meaning of a “carry-over effect” and how it 
can be minimized. 

g. Briefly describe random assignment and its advantages. 

h. Discuss the types, and importance, of blinding in a study. 

i. Discuss how “unblinding” could occur in a study. 

6. Section 6 — Treatment Considerations 
Drug Considerations Subsection 

a. Briefly describe the following points in an efficacy study: 

1. Related to dosage considerations: 

1) relationship between dose and response 

2) whether potency implies a therapeutic advan- 
tage 

3) potential problems with fixed doses 

4) importance of therapeutic ranges 

5) active control dosage 

6) equivalency of dosage range 

2. Related to dosage form considerations: 

1) convenience/ease of administration 

2) absorption/bioavailability 

3) time to achieve therapeutic concentrations 

3. Related to dosage regimen considerations: 

1) timing in relation to food 

2) timing during day 

3) duration of administration 

4. Related to drug concentrations: 

1) when drug concentrations should be monitored 

2) steady-state concentrations 

5. Related to concurrent medication: 

1) interaction with study drug(s) 

2) effect on disease state 

3) amount taken 

6. Related to adverse effects: 

1) reporting of cause and effect relationship 
(memorization of definitions is not required) 

2) determination of intensity 

Study Site or Setting Subsection 

a. Indicate whether studies performed using patients in one 
type of setting can be extrapolated to another type. 

b. Differentiate the study features/factors which are charac- 
teristic of inpatient vs. outpatient settings. 

c. Briefly describe the primary use for artificial study set- 
tings. 

Patient Factors Subsection 

a. Briefly summarize the importance of examining the in- 
clusion and exclusion criteria in a study. 

b. Discuss the importance of the risk-to-benefit ratio when 
analyzing efficacy studies. 

c. Briefly indicate how disease severity can affect compli- 
ance. 

d. Briefly describe whether patient compliance is a concern 
in clinical studies and whether it would be a greater 
problem with inpatients or outpatients. 

e. Explain the meaning and significance of compliance bias. 



f. Recognize the different methods for assessing compli- 
ance and indicate the best way to measure and assess 
compliance in a study. 

g. Briefly describe the intent-to-treat and exclusion of sub- 
jects approaches to analyzing data which includes 
noncompliant patients. 

7. Section 7 — Measurements 

a. Briefly describe random error and systematic error and 
what can be done to minimize each type. 

b. Explain the meaning of validity and differentiate be- 
tween internal and external validity. 

c. Describe the meaning of reliability, sensitivity, and speci- 
ficity. 

d . Discuss the importance of evaluating whether false posi- 
tive or false negative results occurred in a study. 

e. Indicate when the endpoints used for efficacy measure- 
ment should be identified or selected for a study. 

f. Describe the four points to consider when evaluating 
whether the measures of outcome used in a study are 
appropriate. 

g. Briefly discuss what the Hawthorne effect refers to. 

8. Section 8 — Statistics 

Subsection 1 — Variables, Data and Distributions 

a. Differentiate between the dependent and independent 
variables in a clinical study. 

b. Describe nominal, ordinal, and continuous scales (levels) 
of measurement. 

c. Explain what a histogram is and what it is used for. 

d. Describe a normal distribution, an asymmetric distribu- 
tion, and the terms for data distributions with twoormore 
peaks. 

Subsection 2 — Measures of Central Tendency 

a. Describe the mean and the level (s) of data for which it can 
be appropriately used. 

b. Describe the median and the level(s) of data for which it 
can be appropriately used. 

c. Discuss whether the mean and median values can be 
significantly affected by outlying data points. 

d. Describe the mode and the level(s) of data for which it 
can be appropriately used. 

e. For each of the following data distributions, describe the 
relationship between the mean, median, and mode: 

1) Normal distribution 

2) Symmetrical distribution with two peaks 

3) Asymmetric distribution with one peak 

4) Asymmetric and bimodal distribution 

Subsection 3— Measures of Variability 

a. Explain the meaning of the range and whether it can be 
affected by outlying data. 

b. Describe standard deviation, the types of data it is used 
for, and the % of data points included in the mean ±1SD, 
the mean ± 2 SD, and the mean ± 3 SD. 

c. Describe how the size of the SD influences the spread of 
data points around the mean. 

d. Define the term variance. 

e. Describe the standard error of the mean and its relation- 
ship to the SD. 

f. Discuss the meaning of the confidence interval. 

g. Describe how the width of the Cl varies depending on: 

1) the level of confidence selected, e.g., 90% Civs. 95% 
Cl vs. 99% a 

2) the SD of the study sample 

3) the sample size (i.e., size of study group) 

h. For each level (scale) of measurement, indicate whether 
the SD, SEM, and Cl can be appropriately used to 
describe it. 
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Subsection 4 — Statistical Inference 

a. Differentiate between a population and a sample, and a 
parameter and a statistic. 

b. Describe the purpose of hypothesis testing and statistical 
inference. 

c. Explain the difference between Type I and Type II errors 
during hypothesis testing. 

d. Describe what the terms alpha and beta refer to in regard 
to error, and indicate the usual numerical cut-off for an 
acceptable level of alpha and beta in clinical studies. 

e. Describe how Type II error can be minimized in a study. 

f. Describe the meaning of a probability (P) value and its 
normal cut-off for concluding that statistical significance 
exists. 

g. Describe how a P value larger than the normal cut-off 
would be interpreted. 

h. Discuss the meaning of statistical power and indicate how 
it is mathematically defined. 

i. Describe three factors which affect the power of a statis- 
tical test. 

j. Indicate when it would be appropriate to use one-tailed 
vs. two-tailed statistical tests. 

k. Indicate whether P values or confidence intervals reveal 
the size of differences between study groups. 

l. Differentiate between clinical and statistical significance 
in a study. 

Subsection 5 — Parametric Tests 

a. Describe the differences between the assumptions neces- 
sary for the use of parametric tests vs. nonparametric 
tests. 

b. Describe the purpose of a /-test and differentiate between 
paired and nonpaired /-tests. 

c. Determine whether a t-test reported in a clinical study 
was appropriately used. 

d. Describe the problem with the use of multiple t-tests for 
comparisons between more than two groups. 

e. Describe the purpose of an ANOVA procedure and the 
assumptions which apply for its use. 

f . Determine whether an ANOVA procedure reported in a 
clinical study was appropriately used. 

g. Describe the meaning of an F-ratio from an ANOVA 
procedure. 

h. Discuss when it would be appropriate to use a one-way 
ANOVA, a two-way ANOVA, and a repeated measures 
ANOVA. 

i. Recognize the types of multiple comparison procedures 
and explain the purpose for their use. 

Subsection 6 — Nonparametric Tests 

a. Describe when it is appropriate to use nonparametric 
tests. 

b. Indicate the level of data for which it is appropriate to use 
the Chi-Square test, Fisher’s exact test, McNemar test, 
and Rows by Columns test 

c. Briefly explain the theory behind the Chi-Square test. 

d. List the assumptions necessary for the use of the Chi- 
Square test. 

e. Briefly explain when the Fisher’s exact test should be 
used instead of the Chi-Square test. 

f. Describe when it would be appropriate to use the 
McNemar test. 

g. Briefly indicate when and why a corrected Chi-Square 
test (Y ate’s correction) is used. 

h. Describe when it would be appropriate to use the Mann- 
Whitney U (Wilcoxon rank -sum) test. 

i. Describe when it would be appropriate to use the Wilcoxon 
signed-rank test. 

j. Discuss when it would be appropriate to use the Kruskal- 
Wallis test, and indicate the parametric test which would 
be its counterpart. 



k . Discuss when it would be appropriate to use the Frieciman 
test, and indicate the parametric test which would be its 
counterpart. 

Subsection 7 — Correlation 

a. Describe the purpose of performing a correlation analy- 
sis. 

b. Discuss the correlation coefficient (r) in regard to its 
range of values and interpretation. 

c. Identify two methods used to calculate correlation coef- 
ficients and briefly differentiate between the criteria for 
their appropriate use. 

d. Discuss how large an r value should be to provide a 
meaningful result. 

e. Describe how to predict the variability in one measure- 
ment which can be accounted for by the other in a 
correlation analysis. 

f. Describe whether or not correlations can be tested for 
statistical significance. 

g. Explain whether a statistically significant r implies that a 
strong or important correlation exists. 

9. Section 9 — Data Handling 

a. Describe the significance of patient drop-outs from a 
study. 

b. Discuss the advantages and disadvantages of two meth- 
ods used for the analysis of data, the intent-to-treat and 
the exclusion of subjects methods. 

c. Explain how investigators should handle drop-outs in 
their published studies. 

d. Explain whether or not missing data points are a signifi- 
cant problem in a clinical study. 

e. Discuss why it is important for a description of variability 
to be included with any summary data. 

f. Describe the significance of a small vs. large SD in a 
dataset. 

g. Describe two major points to consider when evaluating 
the outcome data reported in a study. 

h. Briefly describe four characteristics which tables and 
graphs in a published study should have. 

i. Indicate how one should ideally include the zero point in 
a graph that has many data points of large value. 

j. Indicate a circumstance in which percentages can be 
misleading. 

k. Discuss whether a reported percent change in a value is 
useful by itself in a study. 

l. Describe what to look for when determining whether 
statistical tests or procedures were appropriately used in 
a study. 



APPENDIX B. QUESTIONS FROM PRETEST AND 
POSTTEST 

1. In a study to determine whether beta- agonists increased the 
risk of near death in patients with asthma, 150 patients who 
had a near-fatal asthma episode within the previous two years 
and 200 patients without such an episode were selected. 
Patients* records were examined to determine their drug use. 
Beta-agonists were used twice as frequently in the patients 
who had a near-fatal asthma episode. This study is best 
described as a: 

a. controlled experiment 

b. cohort study 

c. descriptive study 

d. case-control study 

e. cross-sectional study 

2. Explain in your own words the meaning of power as indicated 
in this sentence from pg. 166 of the article you were given to 
read, “Statistically, it was determined that 23 subjects would 
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* be needed to detect a difference between the treatments with 
a power of 90 and a level of significance of .10.” 

3. Briefly describe why a nonparametric test was utilized for the 
intensity rating scale ( Le ., rating of the severity of the reac- 
tions) data. 

4. a. Describe what a Type I error refers to. 

b. What is another term for a Type I error level? 

5. A study reports that a statistically significant difference exists 
between two antihypertensive agents. Does this imply that 
clinical significance also exists? Explain. 

6. Which type of blinding is preferred for drug efficacy studies? 

a. Unblinded 

b. Single-blind 

c. Double-blind 



d. All are equally appropriate 

7. In a study comparing the effectiveness of clonidine to 
propranolol for the treatment of mild hypertension, half of the 
patients are randomized to receive clonidine first, and the 
other half propranolol. After two months, the patients receiv- 
ing clonidine are given propranolol and those receiving 
propranolol are given clonidine. 

The type of design used in this study is: 

a. “before and after” or time series 

b. concurrent control or parallel treatment 

c. cross-over 

8. Based on the results from the study you were given to read, 
would it be appropriate to conclude that any smoker would 
significantly benefit from use of the transderm al patch? Ex- 
plain. 
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Appendix D 



Computer Program Evaluation Su mm ary 
Additional Year of Evaluation - 1994 



Rating scale used: 5 = Strongly agree, 4 = Agree, 3 = Neutral, 2 = Disagree, 1 = Strongly disagree 

1 . Appropriate number of topics covered in computer program 

5,4 = 54 students, 3=9 students, 2,1= 11 students 

2. Already knew most of information in computer program 

5,4 = 3 students, 3=2 students, 2 , 1 = 70 students 

3 . Examples in computer program helped me understand information 

5,4 = 56 students, 3 = 12 students, 2,1 = 7 students 

4. Learned new information from the computer program 

5 , 4 = 63 students, 3 = 6 students, 2,1=6 students 

5. Learned useful and relevant information* 

5 , 4 = 33 students, 3 = 21 students, 2,1 = 21 students 



6. Liked colors used in program 

5 , 4 = 56 students, 3 = 15 students, 2,1=4 students 

7. Liked screen design and way information presented 

5 , 4 = 54 students, 3=18 students, 2,1=3 students 

8. Difficult to understand material in computer program* 

5,4 = 27 students, 3 = 19 students, 2, 1 = 29 students 

9. Computer program workbook was helpful 

5 , 4 = 47 students, 3 = 15 students, 2,1=13 students 

10. Appropriate amount of time allocated for computer program* 
5,4=18 students, 3 = 10 students, 2 , 1 = 47 students 

1 1 . Program improved ability to read and evaluate published drug studies 

5 , 4 = 47 students, 3 = 16 students, 2,1=12 students 

12. Information in program geared to appropriate level 

5 , 4 = 47 students, 3 = 14 students, 2,1=14 students 

[ 3 . Prefer to use computer rather than attend lectures* 

5^4 — 26 students, 3 = 15 students, 2,1= 33 students 

4. Overall quality of program was high 

5 , 4 = 44 students, 3 = 23 students, 2,1=8 students 



ALo TvSh t T P it in i u 3Sthat the COl ! rse st,U re q uired to ° much time for a 2 credit hour class; 
no thi n- Student * ' disliked having to spend a great deal of time in the computer based learning center 

S * ?f P ;?u § f m ’ u nd P referred ? s »mply attend a lecture and receive handout materials; some students 
did nut feel tnai a pharmacist needs to know how to evaluate drug studies (although it is clear to 

v^r C ^°tS" atl< 5T de that ?* S Sld11 is n0t only im P ortan t, but a specific objective of the newly proposed 6 
year entry .evel pharmacy degree program); some students expressed difficulty in understanding some of 

£cS“„H rPro8 , ra ?,. matenal \ eSpeCially ,he S,a,istics sec!ions al1 st ““M=s had no statistics^ 

Dackground pnor to this course). 
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Appendix E 
Information for FIPSE 



The investigators found the FIPSE staff and program officers to be 
extremely helpful in all aspects of the project. They were available to answer any 
quesnons and alway 8 did so in a timely manner. When mistakes were made on 
S i p U g f (and th fre were some!), our program officer Dr. David Johnson was 
polite, tolerant and informative. He worked with us to resolve problems. Of the 
various granting agencies that the investigators have dealt with in the past FIPSE 
was truly unique with regard to the efforts expended to work with tire 
investigators. The annual project directors’ meeting is an excellent idea that 
needs to be continued. These meetings were a true learning experience. It was 
very beneficial for the various project directors to speak with each other, share 
common educational experiences (across a variety of diverse disciplines), and 
learn what each other is doing. 



We feel that FIPSE should continue to provide funding consideration for 
computer based education projects such as ours. How these types of materials can 
be used optimally in the curriculum is an area that still needs to be fully explored 
Also, distance learning and nontraditional programs (i.e., off-campus, part-time) * 
are emerging areas of interest in the medical professions. The optimal use of the 
computer as a learning tool in these environments should be studied. 



Finally, we wish to personally thank the FIPSE staff for their support of 
our idea and our program officers (Dr. Eulalia Cobb initially, followed by Dr. 
David Johnson) for helping us to make it a reality. It is appreciated 
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